Data-driven Economic NMPC using Reinforcement Learning

Reinforcement Learning (RL) is a powerful tool to perform data-driven optimal control without relying on a model of the system. However, RL struggles to provide hard guarantees on the behavior of the resulting control scheme. In contrast, Nonlinear Model Predictive Control (NMPC) and Economic NMPC (ENMPC) are standard tools for the closed-loop optimal control of complex systems with constraints and limitations, and benefit from a rich theory to assess their closed-loop behavior. Unfortunately, the performance of (E)NMPC hinges on the quality of the model underlying the control scheme. In this paper, we show that an (E)NMPC scheme can be tuned to deliver the optimal policy of the real system even when using a wrong model. This result also holds for real systems having stochastic dynamics. This entails that ENMPC can be used as a new type of function approximator within RL. Furthermore, we investigate our results in the context of ENMPC and formally connect them to the concept of dissipativity, which is central for the ENMPC stability. Finally, we detail how these results can be used to deploy classic RL tools for tuning (E)NMPC schemes. We apply these tools on both a classical linear MPC setting and a standard nonlinear example from the ENMPC literature

An Economic NMPC Formulation for Wind Turbine Control

Model Predictive Control (MPC) is a strong candidate for the control of large Multi-MegaWatt Wind Turbine Generators. Several MPC and some Nonlinear MPC scheme have been proposed in the literature, based on reference-tracking objective functions. While the resulting schemes offer very promising results, the difficulty of tuning a reference-tracking NMPC scheme for performance is likely to be a hindrance to the industrial success of NMPC-based WTG control. Because they directly maximize the system performance, economic NMPC schemes are more straightforward to tune. Economic NMPC schemes present, however, some known difficulties that are a serious obstacle to their real-time deployment. This paper presents an economic NMPC formulation for maximizing the generated power of wind turbine generators, which does not suffer from such difficulties. The relationship between the proposed and more classical reference-tracking approaches is formally established

Reinforcement Learning Based on Real-Time Iteration NMPC

Reinforcement Learning (RL) has proven a stunning ability to learn optimal policies from data without any prior knowledge on the process. The main drawback of RL is that it is typically very difficult to guarantee stability and safety. On the other hand, Nonlinear Model Predictive Control (NMPC) is an advanced model-based control technique which does guarantee safety and stability, but only yields optimality for the nominal model. Therefore, it has been recently proposed to use NMPC as a function approximator within RL. While the ability of this approach to yield good performance has been demonstrated, the main drawback hindering its applicability is related to the computational burden of NMPC, which has to be solved to full convergence. In practice, however, computationally efficient algorithms such as the Real-Time Iteration (RTI) scheme are deployed in order to return an approximate NMPC solution in very short time. In this paper we bridge this gap by extending the existing theoretical framework to also cover RL based on RTI NMPC. We demonstrate the effectiveness of this new RL approach with a nontrivial example modeling a challenging nonlinear system subject to stochastic perturbations with the objective of optimizing an economic cost.Comment: accepted for the IFAC World Congress 202

A Primal-Dual Newton Method for Distributed Quadratic Programming

This paper considers the problem of solving Quadratic Programs (QP) arising in the context of distributed optimization and optimal control. A dual decomposition approach is used, where the problem is decomposed and solved in parallel, while the coupling constraints are enforced via manipulating the dual variables. In this paper, the local problems are solved using a primal-dual interior point method and the dual variables are updated using a Newton iteration, providing a fast convergence rate. Linear predictors for the local primaldual variables and the dual variables are introduced to help the convergence of the algorithm. We observe a fast and consistent practical convergence for the proposed algorithm

Economic MPC of Markov Decision Processes: Dissipativity in Undiscounted Infinite-Horizon Optimal Control

Economic Model Predictive Control (MPC) dissipativity theory is central to discussing the stability of policies resulting from minimizing economic stage costs. In its current form, the dissipativity theory for economic MPC applies to problems based on deterministic dynamics or to very specific classes of stochastic problems, and does not readily extend to generic Markov Decision Processes. In this paper, we clarify the core reason for this difficulty, and propose a generalization of the economic MPC dissipativity theory that circumvents it. This generalization focuses on undiscounted infinite-horizon problems and is based on nonlinear stage cost functionals, allowing one to discuss the Lyapunov asymptotic stability of policies for Markov Decision Processes in terms of the probability measures underlying their stochastic dynamics. This theory is illustrated for the stochastic Linear Quadratic Regulator with Gaussian process noise, for which a storage functional can be provided explicitly. For the sake of brevity, we limit our discussion to undiscounted Markov Decision Processes

A Real-time MHE and NMPC Scheme for Wind Turbine Control

Among the several problems arising in the Airborne Wind Energy paradigm, an essential one is the control of the tethered airfoil trajectory during power generation. Tethered flight is a fast, strongly nonlinear, unstable and constrained process, motivating control approaches based on fast Non-linear Model Predictive Control. In this paper, a computationally efficient model is proposed, based on Differential-Algebraic equations. A control scheme based on Nonlinear Model Predictive Control (NMPC) and an estimator based on Moving Horizon Estimation (MHE) is proposed to handle the wind turbulences. In order to make a real-time application of Non-linear Model Predictive Control possible, a Real-Time Iteration scheme is proposed

Airborne Wind Energy: Airfoil-Airmass Interaction

The Airborne Wind Energy paradigm proposes to generate energy by flying a tethered airfoil across the wind flow at a high velocity. While Airborne Wind Energy enables flight in higher-altitude, stronger wind layers, the extra drag generated by the tether motion imposes a significant limit to the overall system efficiency. To address this issue, two airfoils with a shared tether can reduce overall system drag. A study proposed in Zanon et al. (2013) confirms this claim by showing that, in the considered scenario, the dual-airfoil system is more advantageous than the single-airfoil one. The results computed in Zanon et al. (2013) however, do not model the interaction between the airfoils and the airmass. In this paper, the impact of the airfoil-airmass interaction on the extracted power is studied. As this phenomenon is complex to model, a blade element-momentum approach is proposed and the problem is solved by means of optimal control techniques

Integrated Charging Scheduling and Operational Control for an Electric Bus Network

The last few years have seen the massive deployment of electric buses in many existing transit networks. However, the planning and operation of an electric bus system differ from that of a bus system with conventional vehicles, and some key problems have not yet been studied in the literature. In this work, we address the integrated operational control and charging scheduling problem for a network of electric buses with a limited opportunity charging capacity. We propose a hierarchical control framework to solve this problem, where the charging and operational decisions are taken jointly by solving a mixed-integer linear program in the high-level control layer. Since this optimization problem might become very large as more bus lines are considered, we propose to apply Lagrangian relaxation in such a way as to exploit the structure of the problem and enable a decomposition into independent subproblems. A local search heuristic is then deployed in order to generate good feasible solutions to the original problem. This entire Lagrangian heuristic procedure is shown to scale much better on transit networks with an increasing number of bus lines than trying to solve the original problem with an off-the-shelf solver. The proposed procedure is then tested in the high-fidelity microscopic traffic environment Vissim on a bus network constructed from an openly available dataset of the city of Chicago. The results show the benefits of combining the charging scheduling decisions together with the real-time operational control of the vehicles as the proposed control framework manages to achieve both a better level of service and lower charging costs over control baselines with predetermined charging schedules.Comment: 29 pages, 9 figure